{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "W10_Tutorial1.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Nn57geGyp2hO"
      },
      "source": [
        "# CIS-522 Week 10 Part 1 \n",
        "# Introduction to Natural Language Processing (NLP)\n",
        "\n",
        "\n",
        "**Instructor:** Lyle Ungar\n",
        "\n",
        "**Content Creators:** Sanjeevini  Ganni, Ameet Rahane, Byron Galbraith\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ymkujLBlqM6F"
      },
      "source": [
        "##Tutorial Objectives\n",
        "\n",
        "\n",
        "(1) Recognize NLP tasks: IR/search,  Question Answering/text completion, MT \\\\\n",
        "(2) Understand distributional similarity on words and context including Context-oblivious embeddings (word2vec, glove, fastText) and multilingual embeddings  \\\\\n",
        "(3) Attention  \\\\\n",
        "(4) Context-sensitive embeddings: BERT and transformers (masking and self-attention) \\\\\n",
        "(5) The many flavors of BERT:  RoBERTa and DistilBERT \\\\\n",
        "(6) Fine-tuning language embeddings \\\\\n",
        "(7) Explaining NLP models \\\\\n",
        "(8) Big language models: GPT-3 and friends  \\\\\n",
        "(9) Societal: Bias in language embeddings \\\\\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VC2Bqp7M12U1"
      },
      "source": [
        "---\n",
        "## Preface\n",
        "We recommend saving this notebook in your Google Drive (`File -> Save a copy in Drive`)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LsvDRvFG15Rw",
        "cellView": "form"
      },
      "source": [
        "#@markdown What is your Pennkey and pod? (text, not numbers, e.g. bfranklin)\n",
        "my_pennkey = '' #@param {type:\"string\"}\n",
        "my_pod = 'Select' #@param ['Select', 'euclidean-wombat', 'sublime-newt', 'buoyant-unicorn', 'lackadaisical-manatee','indelible-stingray','superfluous-lyrebird','discreet-reindeer','quizzical-goldfish','ubiquitous-cheetah','nonchalant-crocodile','fashionable-lemur','spiffy-eagle','electric-emu','quotidian-lion','astute-jellyfish', 'quantum-herring']\n",
        "\n",
        "# start timing\n",
        "import time\n",
        "try:t0;\n",
        "except NameError: t0 = time.time()\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-hN3ZmTOqY86"
      },
      "source": [
        "## Recap the experience from last week"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "cellView": "form",
        "id": "fhsZ1elwT7mB"
      },
      "source": [
        "#@markdown Tell us your thoughts about what you have learned last week. What questions do you have? Discuss with your pod!\n",
        "last_week_recap = '' #@param {type:\"string\"}"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2_oC9dNZqQu-"
      },
      "source": [
        "##Setup"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0RcjygJ2CRSG",
        "cellView": "form"
      },
      "source": [
        "#@title Install\n",
        "!pip install torchtext==0.4.0\n",
        "!git clone https://github.com/facebookresearch/fastText.git\n",
        "%cd fastText\n",
        "!pip install . \n",
        "%cd .."
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "52U7jyR4pngL",
        "cellView": "form"
      },
      "source": [
        "#@title Imports and Seed\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "import time\n",
        "import matplotlib.pyplot as plt\n",
        "import matplotlib.cm as cm\n",
        "% matplotlib inline\n",
        "import re\n",
        "from IPython.display import Image\n",
        "import os\n",
        "from tqdm import tqdm_notebook as tqdm\n",
        "import sys\n",
        "import random\n",
        "import torch\n",
        "import torch.nn as nn\n",
        "from torch.nn import functional as F\n",
        "import torch.optim as optim\n",
        "from torch.autograd import Variable\n",
        "from torchtext import data, datasets\n",
        "from torchtext.vocab import Vectors, FastText\n",
        "import fasttext\n",
        "import requests\n",
        "import zipfile\n",
        "import nltk\n",
        "nltk.download('punkt')\n",
        "from nltk.tokenize import word_tokenize\n",
        "\n",
        "seed = 522\n",
        "random.seed(seed)\n",
        "torch.manual_seed(seed)\n",
        "torch.cuda.manual_seed_all(seed)\n",
        "torch.cuda.manual_seed(seed)\n",
        "\n",
        "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
        "print(device)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "l82sJ8BNmty-",
        "cellView": "form"
      },
      "source": [
        "# @title Figure Settings\n",
        "import ipywidgets as widgets\n",
        "%matplotlib inline \n",
        "fig_w, fig_h = (8, 6)\n",
        "plt.rcParams.update({'figure.figsize': (fig_w, fig_h)})\n",
        "%config InlineBackend.figure_format = 'retina'\n",
        "SMALL_SIZE = 12\n",
        "\n",
        "\n",
        "plt.style.use(\"https://raw.githubusercontent.com/NeuromatchAcademy/\"\n",
        "              \"course-content/master/nma.mplstyle\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ik8be94eRLPt",
        "cellView": "form"
      },
      "source": [
        "#@title Helper functions\n",
        "def cosine_similarity(vec_a, vec_b):\n",
        "        \"\"\"Compute cosine similarity between vec_a and vec_b\"\"\"\n",
        "        return np.dot(vec_a, vec_b) / \\\n",
        "            (np.linalg.norm(vec_a) * np.linalg.norm(vec_b))\n",
        "def tokenize(sentences):\n",
        "  #Tokenize the sentence\n",
        "  #from nltk.tokenize library use word_tokenize\n",
        "  token = word_tokenize(sentences)\n",
        "  \n",
        "  return token\n",
        "\n",
        "def plot_train_val(x, train, val, train_label, val_label, title):\n",
        "  plt.plot(x, train, label=train_label)\n",
        "  plt.plot(x, val, label=val_label)\n",
        "  plt.legend(loc='lower right')\n",
        "  plt.xlabel('epoch')\n",
        "  plt.title(title)\n",
        "  plt.show()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Bi_7aepGAZkf"
      },
      "source": [
        "###Data"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xCZ56QJ96aMW",
        "cellView": "form"
      },
      "source": [
        "#@title Load Data\n",
        "\n",
        "def load_dataset(emb_vectors, sentence_length = 50):\n",
        "    TEXT = data.Field(sequential=True, tokenize=tokenize, lower=True, include_lengths=True, batch_first=True, fix_length=sentence_length)\n",
        "    LABEL = data.LabelField(dtype=torch.float)\n",
        "\n",
        "    train_data, test_data = datasets.IMDB.splits(TEXT, LABEL)\n",
        "\n",
        "    TEXT.build_vocab(train_data, vectors=emb_vectors)\n",
        "    LABEL.build_vocab(train_data)\n",
        "\n",
        "    train_data, valid_data = train_data.split(split_ratio=0.7, random_state = random.seed(seed))\n",
        "    train_iter, valid_iter, test_iter = data.BucketIterator.splits((train_data, valid_data, test_data), batch_size=32, sort_key=lambda x: len(x.text), repeat=False, shuffle=True)\n",
        "    vocab_size = len(TEXT.vocab)\n",
        "\n",
        "    return TEXT, vocab_size, train_iter, valid_iter, test_iter\n",
        "\n",
        "\n",
        "def download_file_from_google_drive(id, destination):\n",
        "    URL = \"https://docs.google.com/uc?export=download\"\n",
        "\n",
        "    session = requests.Session()\n",
        "\n",
        "    response = session.get(URL, params = { 'id' : id }, stream = True)\n",
        "    token = get_confirm_token(response)\n",
        "\n",
        "    if token:\n",
        "        params = { 'id' : id, 'confirm' : token }\n",
        "        response = session.get(URL, params = params, stream = True)\n",
        "\n",
        "    save_response_content(response, destination)    \n",
        "\n",
        "def get_confirm_token(response):\n",
        "    for key, value in response.cookies.items():\n",
        "        if key.startswith('download_warning'):\n",
        "            return value\n",
        "\n",
        "    return None\n",
        "\n",
        "def save_response_content(response, destination):\n",
        "    CHUNK_SIZE = 32768\n",
        "\n",
        "    with open(destination, \"wb\") as f:\n",
        "        for chunk in response.iter_content(CHUNK_SIZE):\n",
        "            if chunk: # filter out keep-alive new chunks\n",
        "                f.write(chunk)\n",
        "\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FfNWcsbcVR1Z"
      },
      "source": [
        "### Introduction\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "opCHUMupREeA",
        "cellView": "form"
      },
      "source": [
        "#@title Video : NLP  Introduction\n",
        "from IPython.display import YouTubeVideo\n",
        "video = YouTubeVideo(id=\"vzsDA8qazW8\", width=854, height=480, fs=1)\n",
        "print(\"Video available at https://youtube.com/watch?v=\" + video.id)\n",
        "\n",
        "import time\n",
        "try: t0;\n",
        "except NameError: t0=time.time()\n",
        "\n",
        "video"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2i6C9ZOaQTbQ"
      },
      "source": [
        "*Estimated time: 15 minutes since start*"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cMcTWxhcqwIf"
      },
      "source": [
        "---\n",
        "##Section 1: Overview of NLP\n",
        "\n",
        "Natural Language Processing (NLP) techniques process and analyze large amounts of natural language data. \n",
        "NLP applications include:\n",
        "\n",
        "1.   Sentiment Analysis: Understanding the views of people/customers towards a specific product or service. \n",
        "2.   Virtual Assistants: Alexa, Siri and Google Assistant, etc.\n",
        "3.   Machine Translation: Translating speech or text from one language to another.  \n",
        "4.   Information retrieval and search engines\n",
        "5.   Question Answering\n",
        "6.   Summarization\n",
        "7.   Identifying Fake News and spam: Ironically, NLP can also be used to generate fake news.\n",
        "\n",
        "NLP is also increasingly being used in recruitment decisions, litigation tasks, etc.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "cellView": "form",
        "id": "6b1ZUm25Vhx_"
      },
      "source": [
        "#@title Video : Language Processing\n",
        "from IPython.display import YouTubeVideo\n",
        "video = YouTubeVideo(id=\"TZv5NmAjUyA\", width=854, height=480, fs=1)\n",
        "print(\"Video available at https://youtube.com/watch?v=\" + video.id)\n",
        "\n",
        "import time\n",
        "try: t0;\n",
        "except NameError: t0=time.time()\n",
        "\n",
        "video"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0rI0Kp-zg5_a"
      },
      "source": [
        "###Question"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "D019t0R-qzFp",
        "cellView": "form"
      },
      "source": [
        "#report to Airtable\n",
        "#@markdown What's your favorite NLP application?  What are you most excited to learn about? Discuss with your pod!\n",
        "NLP_summary = '' #@param {type:\"string\"}"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "13WZESEaQo_7"
      },
      "source": [
        "*Estimated time: 35 minutes since start*"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eiXOXHPD7UWZ"
      },
      "source": [
        "---\n",
        "##Section 2: Word Embeddings"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "m_pGGV5CRG_t",
        "cellView": "form"
      },
      "source": [
        "#@title Video : Embeddings\n",
        "from IPython.display import YouTubeVideo\n",
        "video = YouTubeVideo(id=\"ufqGdQcwa5A\", width=854, height=480, fs=1)\n",
        "print(\"Video available at https://youtube.com/watch?v=\" + video.id)\n",
        "\n",
        "import time\n",
        "try: t0;\n",
        "except NameError: t0=time.time()\n",
        "\n",
        "video"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LkU-9UT1c-4G"
      },
      "source": [
        "\n",
        "Words or subword units such as morphemes are the basic units that we use to express meaning  in language. The technique of mapping words to vectors of real numbers is known as word embedding. \n",
        "\n",
        "Last week, we saw that one-hot word vectors cannot accurately express the similarity between different words. The cosine similarity between the one-hot vectors of any two different words is zero.\n",
        "\n",
        "This week, we will be using pretrained fastText embeddings, a context-oblivious embedding similar to word2vec. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9F7FT7dkdP5t"
      },
      "source": [
        "###Exercise 1: Embedding Manipulation"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lWctWkpUA7rX"
      },
      "source": [
        "Let's use the FastText library to manipulate the embeddings. \n",
        "First, find the embedding for the word \"King\""
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "T8jn3QasSGfg"
      },
      "source": [
        "#@title Download FastText English Embeddings of dimension 100\n",
        "\n",
        "# 1d2fzekxTgDIA9LEYskPXT1sfccgAGmUF\n",
        "# file_id = '11qRoFOUiGnvM04ERcN_6Qn6O_dJ2KPLk'\n",
        "# destination = 'cc.en.100.bin.zip'\n",
        "# download_file_from_google_drive(file_id, destination)\n",
        "!wget -O cc.en.100.bin.gz https://osf.io/2frqg/download\n",
        "\n",
        "\n",
        "import os\n",
        "os.listdir('.')\n",
        "with zipfile.ZipFile('cc.en.100.bin.gz', 'r') as zip_ref:\n",
        "    zip_ref.extractall('.')\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dVXvcxJ5-vMI"
      },
      "source": [
        "#####Load 100 dimension FastText Vectors using FastText library\n",
        "ft_en_vectors = fasttext.load_model('cc.en.100.bin')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KI1sJp4Pd9eg"
      },
      "source": [
        "print(\"Length of the embedding is: \", len(ft_en_vectors.get_word_vector('king')))\n",
        "print(\"Embedding for the word King is: \", ft_en_vectors.get_word_vector('king'))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vuzeZR7pFT9E"
      },
      "source": [
        "Cosine similarity is used for similarities between words. Similarity is a scalar between 0 and 1."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-g35ame9HCRV"
      },
      "source": [
        "Now find the 10 most similar words to \"King\""
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HaRMMjIoGDsr"
      },
      "source": [
        "ft_en_vectors.get_nearest_neighbors(\"king\",10) # Most similar by key"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "n35pqYiobxaJ"
      },
      "source": [
        "###Exercise 2: Word Similarity"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xugCyA0uNK4Y"
      },
      "source": [
        "More on similarity between words. Let's check how similar different pairs of word are. Feel free to play around.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "P-G9Iy6RAJ_Q"
      },
      "source": [
        "def getSimilarity(word1, word2):\n",
        "  v1 = ft_en_vectors.get_word_vector(word1)\n",
        "  v2 = ft_en_vectors.get_word_vector(word2)\n",
        "  return cosine_similarity(v1, v2)\n",
        "\n",
        "print(\"Similarity between the words King and Queen: \", getSimilarity(\"king\", \"queen\"))\n",
        "print(\"Similarity between the words King and Knight: \", getSimilarity(\"king\", \"knight\"))\n",
        "print(\"Similarity between the words King and Rock: \", getSimilarity(\"king\", \"rock\"))\n",
        "print(\"Similarity between the words King and Twenty: \", getSimilarity(\"king\", \"twenty\"))\n",
        "print()\n",
        "print(\"Similarity between the words Dog and Cat: \", getSimilarity(\"dog\", \"cat\"))\n",
        "print(\"Similarity between the words Ascending and Descending: \", getSimilarity(\"ascending\", \"descending\"))\n",
        "print(\"Similarity between the words Victory and Defeat: \", getSimilarity(\"victory\", \"defeat\"))\n",
        "print(\"Similarity between the words Less and More: \", getSimilarity(\"less\", \"more\"))\n",
        "print(\"Similarity between the words True and False: \", getSimilarity(\"true\", \"false\"))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xGrL22iWMasA"
      },
      "source": [
        "###Question\n",
        "\n",
        "Why do you think the words with opposite meanings like \"Ascending\" and \"Descending\" or \"Victory\" and \"Defeat\" have high similarity? Discuss with your pod."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MQvX_5TCR7tn",
        "cellView": "form"
      },
      "source": [
        "#report to Airtable\n",
        "words_with_opposite_meanings = '' #@param {type:\"string\"}"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZN4gXHA4HljR"
      },
      "source": [
        "Using embeddings, we can find the words that appear in similar contexts. But, what happens if the word has several different meanings? "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "n1F4QN8GR70q"
      },
      "source": [
        "###Exercise 3: Homonym Similarity"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3dpxElc9cMg7"
      },
      "source": [
        "Find the similarity for homonyms with their different meanings. The first one has been implemented for you."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Gz20ZseyOO0m"
      },
      "source": [
        "#######################     Words with multiple meanings     ##########################\n",
        "print(\"Similarity between the words Cricket and Insect: \", getSimilarity(\"cricket\", \"insect\"))\n",
        "print(\"Similarity between the words Cricket and Sport: \", getSimilarity(\"cricket\", \"sport\"))\n",
        "\n",
        "#Try the same for two more pairs\n",
        "# print(\"Similarity between the words ___ and ___: \", getSimilarity(...))\n",
        "# print(\"Similarity between the words ___ and ___: \", getSimilarity(...))\n",
        "\n",
        "# print(\"Similarity between the words ___ and ___: \", getSimilarity(...))\n",
        "# print(\"Similarity between the words ___ and ___: \", getSimilarity(...))\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DlKxoozWg2uJ"
      },
      "source": [
        "###Word Analogies"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RXkZ-fyKiy5c"
      },
      "source": [
        "Embeddings can be used to find word analogies.\n",
        "Let's try it:\n",
        "1.   Man : Woman  ::  King : _____\n",
        "2.  Germany: Berlin :: France : ______\n",
        "3.  Leaf : Tree  ::  Petal : _____\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fHKt2IZ0GpPB"
      },
      "source": [
        "## Use get_analogies() funnction. The words have to be in the order Positive, negative,  Positve\n",
        "\n",
        "# Man : Woman  ::  King : _____\n",
        "# Positive=(woman, king), Negative=(man)\n",
        "print(ft_en_vectors.get_analogies(\"woman\", \"man\", \"king\",1)) \n",
        "\n",
        "# Germany: Berlin :: France : ______\n",
        "# Positive=(berlin, frannce), Negative=(germany)\n",
        "print(ft_en_vectors.get_analogies(\"berlin\", \"germany\", \"france\",1))\n",
        "\n",
        "# Leaf : Tree  ::  Petal : _____\n",
        "# Positive=(tree, petal), Negative=(leaf)\n",
        "print(ft_en_vectors.get_analogies(\"tree\", \"leaf\", \"petal\",1))\n",
        "\n",
        "# Hammer : Nail  ::  Comb : _____\n",
        "# Positive=(nail, comb), Negative=(hammer)\n",
        "print(ft_en_vectors.get_analogies(\"nail\", \"hammer\", \"comb\",1))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7eZ6ZdHQkop-"
      },
      "source": [
        "But, does it always work?\n",
        "\n",
        "\n",
        "1.   Poverty : Wealth  :: Sickness : _____\n",
        "2.   train : board :: horse : _____"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "M5asyCbMHaQE"
      },
      "source": [
        "# Poverty : Wealth  :: Sickness : _____\n",
        "print(ft_en_vectors.get_analogies(\"wealth\", \"poverty\", \"sickness\",1))\n",
        "\n",
        "# train : board :: horse : _____\n",
        "print(ft_en_vectors.get_analogies(\"board\", \"train\", \"horse\",1))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zVkyyvCzfNVb"
      },
      "source": [
        "#### Question "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZJlyOpC4gqED",
        "cellView": "form"
      },
      "source": [
        "#@markdown What are the drawbacks of word embeddings? How can they be improved?\n",
        "#report to Airtable\n",
        "embeddings_improve = '' #@param {type:\"string\"}"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JmHQUcI42Ozk"
      },
      "source": [
        "###Section 2.1: Multilingual Embeddings\n",
        "\n",
        "Traditionally, word embeddings have been lnaguage-specific, with embeddings for each language trained separately and existing in entirely different vector spaces. But, what if we wanted to compare words in one language to another language? Say, we want to create a text classifier with a corpus consisting of English and Spanish words. \n",
        "\n",
        "We use the multilingual word embeddings provided in fastText. More information can be found [here](https://engineering.fb.com/2018/01/24/ml-applications/under-the-hood-multilingual-embeddings/).\n",
        "\n",
        "#### Training multilingual embeddings\n",
        "We first train separate embeddings for each language using fastText and a combination of data from Facebook and Wikipedia. Then, we find a dictionary of common words between the two languages. The dictionaries are automatically induced from parallel data - datasets that consist of a pair of sentences in two different languages that have the same meaning. \n",
        "\n",
        "Then, we find a matrix that projects the embeddings into a common space between the given languages. The matrix is designed to minimize the distance between a word $x_i$ and its projection $y_i$. If our dictionary consists of pairs $(x_i, y_i)$, our projector $M$ would be: \n",
        "\n",
        "$$M = \\text{argmin}_W\\sum_i ||x_i - Wy_ii||^2$$\n",
        "\n",
        "Also, the projector matrix $W$ is constrained to e orthogonal so that original distances between word embedding vectors are preserved. Multilingual models are trained by using our multilingual word embeddings as the base representations in DeepText and “freezing” them, or leaving them unchanged during the training process. \n",
        "\n",
        "\n",
        "After going through this, try to replicate the above exercises but in different languages! \n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hrTiIoTR2rJG",
        "cellView": "form"
      },
      "source": [
        "#@title Video : Multilingual Embeddings\n",
        "from IPython.display import YouTubeVideo\n",
        "video = YouTubeVideo(id=\"cqNiEsH7Iqc\", width=854, height=480, fs=1)\n",
        "print(\"Video available at https://youtube.com/watch?v=\" + video.id)\n",
        "\n",
        "import time\n",
        "try: t0;\n",
        "except NameError: t0=time.time()\n",
        "\n",
        "video"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QkZDDZryGzJy"
      },
      "source": [
        "#@title Download FastText French Embeddings of dimension 100\n",
        "\n",
        "!wget -O cc.fr.100.bin.gz https://osf.io/6tqvp/download\n",
        "\n",
        "with zipfile.ZipFile('cc.fr.100.bin.gz', 'r') as zip_ref:\n",
        "    zip_ref.extractall('.')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UB-HOFL3hqkI"
      },
      "source": [
        "#####Load 100 dimension FastText Vectors using FastText library\n",
        "french = fasttext.load_model('cc.fr.100.bin')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "niCYe2QllHM2"
      },
      "source": [
        "First, we take a look at the cosine similarity between different languages without projecting them into the same vector space. As you can see, the same words seem to be close to $0$ cosine similarity in different languages - so neither similar nor dissimilar. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TwI-61zSyF6X"
      },
      "source": [
        "hello = ft_en_vectors.get_word_vector('hello')\n",
        "hi = ft_en_vectors.get_word_vector('hi')\n",
        "bonjour = french.get_word_vector('bonjour')\n",
        "\n",
        "print(\"Cosine Similarity between HI and HELLO: {}\".format(cosine_similarity(hello, hi)))\n",
        "print(\"Cosine Similarity between BONJOUR and HELLO: {}\".format(cosine_similarity(hello, bonjour)))\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rnLV8NzfJvUp"
      },
      "source": [
        "cat = ft_en_vectors.get_word_vector('cat')\n",
        "chatte = french.get_word_vector('chatte')\n",
        "chat = french.get_word_vector('chat')\n",
        "\n",
        "print(\"Cosine Similarity between cat and chatte: {}\".format(cosine_similarity(cat, chatte)))\n",
        "print(\"Cosine Similarity between cat and chat: {}\".format(cosine_similarity(cat, chat)))\n",
        "print(\"Cosine Similarity between chatte and chat: {}\".format(cosine_similarity(chatte, chat)))\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Z-4Ywer6lu8l"
      },
      "source": [
        "First, let's define a list of words that are in common between English and French. We'll be using this to make our training matrices. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YoyzY7x-NQn_"
      },
      "source": [
        "en_words = set(ft_en_vectors.words)\n",
        "fr_words = set(french.words)\n",
        "overlap = list(en_words & fr_words)\n",
        "bilingual_dictionary = [(entry, entry) for entry in overlap]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pTUS9JB_l5Jd"
      },
      "source": [
        "We define a few functions to make our lives a bit easier: `make_training_matrices` takes in the source words, target language words, and the set of common words. It then creates a matrix of all the word embeddings of all common words between the languages (in each language). These are our training matrices. \n",
        "\n",
        "The function `learn_transformation` then takes in these matrices, normalizes them, and then performs SVD, which aligns the source language to the target and returns a transformation matrix. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sdF7jqWgNZ3V"
      },
      "source": [
        "def make_training_matrices(source_dictionary, target_dictionary, bilingual_dictionary):\n",
        "    source_matrix = []\n",
        "    target_matrix = []\n",
        "\n",
        "    for (source, target) in tqdm(bilingual_dictionary):\n",
        "        #if source in source_dictionary.words and target in target_dictionary.words:\n",
        "        source_matrix.append(source_dictionary.get_word_vector(source))\n",
        "        target_matrix.append(target_dictionary.get_word_vector(target))\n",
        "\n",
        "    # return training matrices\n",
        "    return np.array(source_matrix), np.array(target_matrix)\n",
        "\n",
        "# from https://stackoverflow.com/questions/21030391/how-to-normalize-array-numpy\n",
        "def normalized(a, axis=-1, order=2):\n",
        "    \"\"\"Utility function to normalize the rows of a numpy array.\"\"\"\n",
        "    l2 = np.atleast_1d(np.linalg.norm(a, order, axis))\n",
        "    l2[l2==0] = 1\n",
        "    return a / np.expand_dims(l2, axis)\n",
        "\n",
        "def learn_transformation(source_matrix, target_matrix, normalize_vectors=True):\n",
        "    \"\"\"\n",
        "    Source and target matrices are numpy arrays, shape\n",
        "    (dictionary_length, embedding_dimension). These contain paired\n",
        "    word vectors from the bilingual dictionary.\n",
        "    \"\"\"\n",
        "    # optionally normalize the training vectors\n",
        "    if normalize_vectors:\n",
        "        source_matrix = normalized(source_matrix)\n",
        "        target_matrix = normalized(target_matrix)\n",
        "\n",
        "    # perform the SVD\n",
        "    product = np.matmul(source_matrix.transpose(), target_matrix)\n",
        "    U, s, V = np.linalg.svd(product)\n",
        "\n",
        "    # return orthogonal transformation which aligns source language to the target\n",
        "    return np.matmul(U, V)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zBoo9SaQmlxD"
      },
      "source": [
        "Now, we just have to put it all together! "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FH6IdDLPQdLJ"
      },
      "source": [
        "source_training_matrix, target_training_matrix = make_training_matrices(ft_en_vectors, french, bilingual_dictionary)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7eU7R44XQtto"
      },
      "source": [
        "transform = learn_transformation(source_training_matrix, target_training_matrix)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "baD77aOVmqVX"
      },
      "source": [
        "Let's run the same examples as above, but this time, whenever we use French words, matrix multiply the embedding by the transpose of the transform matrix. That works a lot better! "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "S7a3L-VOR6ht"
      },
      "source": [
        "hello = ft_en_vectors.get_word_vector('hello')\n",
        "hi = ft_en_vectors.get_word_vector('hi')\n",
        "bonjour = np.matmul(french.get_word_vector('bonjour'), transform.T)\n",
        "\n",
        "print(\"Cosine Similarity between HI and HELLO: {}\".format(cosine_similarity(hello, hi)))\n",
        "print(\"Cosine Similarity between BONJOUR and HELLO: {}\".format(cosine_similarity(hello, bonjour)))\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "a-o5a_DQS6b7"
      },
      "source": [
        "cat = ft_en_vectors.get_word_vector('cat')\n",
        "chatte = np.matmul(french.get_word_vector('chatte'), transform.T)\n",
        "chat = np.matmul(french.get_word_vector('chat'), transform.T)\n",
        "\n",
        "print(\"Cosine Similarity between cat and chatte: {}\".format(cosine_similarity(cat, chatte)))\n",
        "print(\"Cosine Similarity between cat and chat: {}\".format(cosine_similarity(cat, chat)))\n",
        "print(\"Cosine Similarity between chatte and chat: {}\".format(cosine_similarity(chatte, chat)))\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hRYVFi7-m797"
      },
      "source": [
        "Now, try a couple of your own examples. Try some of the examples you looked at in $2.1$, but with English and French. Does it work as expected? "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "AXBTQe_8nMfI"
      },
      "source": [
        "# USE THIS SPACE TO TRY YOUR OWN EXAMPLES"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0uexRXZpkxBI",
        "cellView": "form"
      },
      "source": [
        "#@markdown Clean Up of Previous Files (MUST RUN)\n",
        "!rm cc.en.100.bin.zip\n",
        "!rm cc.fr.100.bin.gz\n",
        "!rm cc.en.100.bin\n",
        "!rm cc.fr.100.bin"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Y40Qxm56hjXr"
      },
      "source": [
        "*Estimated time: 50 minutes since start*"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tcYd3O5gn79L"
      },
      "source": [
        "---\n",
        "##Section 3: Neural Net with word embeddings"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dBoJG26LRKHU",
        "cellView": "form"
      },
      "source": [
        "#@title Sentence Embeddings\n",
        "from IPython.display import YouTubeVideo\n",
        "video = YouTubeVideo(id=\"skFimcn320U\", width=854, height=480, fs=1)\n",
        "print(\"Video available at https://youtube.com/watch?v=\" + video.id)\n",
        "\n",
        "import time\n",
        "try: t0;\n",
        "except NameError: t0=time.time()\n",
        "\n",
        "video"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yjjFcmNlpBfd"
      },
      "source": [
        "As we saw last week, training context-oblivious word embeddings is relatively cheap, but most people still use pre-trained word embeddings. After we cover context sensitive word embeddings we'll see how to \"fine tune\" embeddings (adjust them to the task at hand)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "E0Jfo6R018B3"
      },
      "source": [
        "Let's use the pretrained FastText embeddings to train a neural network on the IMDB dataset from previous week. \n",
        "\n",
        "To recap, the data consists of reviews and sentiments attached to it. It is a binary classification task. Last week we achieved an accuracy of about **70\\%** on the Test data with LSTM network and around **65%** on the Vanilla RNN network.\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fQsw9lNj3PmE"
      },
      "source": [
        "###Exercise 4: Simple feed forward net"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Aj9P4uwtaxh4",
        "cellView": "form"
      },
      "source": [
        "#@title Download embeddings (Run while watching above video)\n",
        "\n",
        "embedding_fasttext = FastText('simple')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4Ez7ZiWXcMeC"
      },
      "source": [
        "This will load 300 dim FastText embeddings. It will take around 2-3 minutes."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MOdb3E0TpzdS"
      },
      "source": [
        "Define a vanilla neural network with linear layers. Then average the word embeddings to get an embedding for the entire review.\n",
        "The neural net will have one hidden layer of size 128."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mHRWikI4PqJ2"
      },
      "source": [
        "class NeuralNet(nn.Module):\n",
        "  def __init__(self, batch_size, output_size, hidden_size, vocab_size, embedding_length, word_embeddings):\n",
        "    super(NeuralNet, self).__init__()\n",
        "\n",
        "    self.batch_size = batch_size\n",
        "    self.output_size = output_size\n",
        "    self.hidden_size = hidden_size\n",
        "    self.vocab_size = vocab_size\n",
        "    self.embedding_length = embedding_length\n",
        "\n",
        "    self.word_embeddings = nn.Embedding(vocab_size, embedding_length)\n",
        "    self.word_embeddings.weight = nn.Parameter(word_embeddings, requires_grad=False)\n",
        "    self.fc1 = nn.Linear(embedding_length, hidden_size)\n",
        "    self.fc2 = nn.Linear(hidden_size, output_size)\n",
        "\n",
        "  def forward(self, inputs):\n",
        "    \n",
        "    input = self.word_embeddings(inputs) #convert text to embeddings\n",
        "    ####################################################################\n",
        "    # Fill in missing code below (...)\n",
        "    raise NotImplementedError(\"Fill in the Neural Net\")\n",
        "    \n",
        "    #Average the word embedddings in a sentence\n",
        "    #Use torch.nn.functional.avg_pool2d to compute the averages\n",
        "    pooled = ...\n",
        "\n",
        "    #Pass the embeddings through the neural net\n",
        "    x = ...\n",
        "    x = ...\n",
        "    output = F.log_softmax(x, dim=1)    \n",
        "    return output\n",
        "\n",
        "  #to_remove solution\n",
        "  def forward(self, inputs):\n",
        "    \n",
        "    input = self.word_embeddings(inputs) #convert text to embeddings\n",
        "    pooled = F.avg_pool2d(input, (input.shape[1], 1)).squeeze(1)\n",
        "\n",
        "    x = self.fc1(pooled)\n",
        "    x = F.relu(x)\n",
        "    x = self.fc2(x)\n",
        "    output = F.log_softmax(x, dim=1)    \n",
        "    return output"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BLuocVQuRObk",
        "cellView": "form"
      },
      "source": [
        "#@title Training and Testing Functions\n",
        "\n",
        "def train(model, device, train_iter, valid_iter, epochs, learning_rate):\n",
        "    criterion = nn.CrossEntropyLoss()\n",
        "    optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)\n",
        "    \n",
        "    train_loss, validation_loss = [], []\n",
        "    train_acc, validation_acc = [], []\n",
        "\n",
        "    for epoch in range(epochs):\n",
        "      #train\n",
        "      model.train()\n",
        "      running_loss = 0.\n",
        "      correct, total = 0, 0 \n",
        "      steps = 0\n",
        "\n",
        "      for idx, batch in enumerate(train_iter):\n",
        "        text = batch.text[0]\n",
        "        target = batch.label\n",
        "        target = torch.autograd.Variable(target).long()\n",
        "        text, target = text.to(device), target.to(device)\n",
        "\n",
        "        # add micro for coding training loop\n",
        "        optimizer.zero_grad()\n",
        "        output = model(text)\n",
        "        loss = criterion(output, target)\n",
        "        loss.backward()\n",
        "        optimizer.step()\n",
        "        steps += 1\n",
        "        running_loss += loss.item()\n",
        "\n",
        "        # get accuracy \n",
        "        _, predicted = torch.max(output, 1)\n",
        "        total += target.size(0)\n",
        "        correct += (predicted == target).sum().item()\n",
        "      train_loss.append(running_loss/len(train_iter))\n",
        "      train_acc.append(correct/total)\n",
        "\n",
        "      print(f'Epoch: {epoch + 1},  Training Loss: {running_loss/len(train_iter):.4f}, Training Accuracy: {100*correct/total: .2f}%')\n",
        "\n",
        "      # evaluate on validation data\n",
        "      model.eval()\n",
        "      running_loss = 0.\n",
        "      correct, total = 0, 0 \n",
        "\n",
        "      with torch.no_grad():\n",
        "        for idx, batch in enumerate(valid_iter):\n",
        "            text = batch.text[0]\n",
        "            target = batch.label\n",
        "            target = torch.autograd.Variable(target).long()\n",
        "            text, target = text.to(device), target.to(device)\n",
        "\n",
        "            optimizer.zero_grad()\n",
        "            output = model(text)\n",
        "    \n",
        "            loss = criterion(output, target)\n",
        "            running_loss += loss.item()\n",
        "\n",
        "            # get accuracy \n",
        "            _, predicted = torch.max(output, 1)\n",
        "            total += target.size(0)\n",
        "            correct += (predicted == target).sum().item()\n",
        "\n",
        "      validation_loss.append(running_loss/len(valid_iter))\n",
        "      validation_acc.append(correct/total)\n",
        "\n",
        "      print (f'Validation Loss: {running_loss/len(valid_iter):.4f}, Validation Accuracy: {100*correct/total: .2f}%')\n",
        "  \n",
        "    return train_loss, train_acc, validation_loss, validation_acc\n",
        "\n",
        "def test(model,  device, test_iter):\n",
        "  model.eval()\n",
        "  correct = 0\n",
        "  total = 0\n",
        "  with torch.no_grad():\n",
        "    for idx, batch in enumerate(test_iter):\n",
        "        text = batch.text[0]\n",
        "        target = batch.label\n",
        "        target = torch.autograd.Variable(target).long()\n",
        "        text, target = text.to(device), target.to(device)\n",
        "\n",
        "        outputs = model(text)\n",
        "        _, predicted = torch.max(outputs, 1)\n",
        "        total += target.size(0)\n",
        "        correct += (predicted == target).sum().item()\n",
        "\n",
        "    acc = 100 * correct / total\n",
        "    return acc"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "F89jVCxRhlRC"
      },
      "source": [
        "##Load the Dataset\n",
        "TEXT, vocab_size, train_iter, valid_iter, test_iter = load_dataset(embedding_fasttext)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "K0ulnNHujG47"
      },
      "source": [
        "learning_rate = 0.0003\n",
        "batch_size = 32\n",
        "output_size = 2\n",
        "hidden_size = 128\n",
        "embedding_length = 300\n",
        "epochs = 15\n",
        "word_embeddings = TEXT.vocab.vectors\n",
        "vocab_size = len(TEXT.vocab)\n",
        "\n",
        "nn_model = NeuralNet(batch_size, output_size, hidden_size, vocab_size, embedding_length, word_embeddings)\n",
        "nn_model.to(device)\n",
        "nn_start_time = time.time()\n",
        "nn_train_loss, nn_train_acc, nn_validation_loss, nn_validation_acc = train(nn_model, device, train_iter, valid_iter, epochs, learning_rate)\n",
        "print(\"--- Time taken to train = %s seconds ---\" % (time.time() - nn_start_time))\n",
        "test_accuracy = test(nn_model, device, test_iter)\n",
        "print('Test Accuracy: ',  test_accuracy, '%')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YXnfP308m6_Z"
      },
      "source": [
        "with plt.xkcd():\n",
        "    plot_train_val(np.arange(0,epochs), nn_train_acc, nn_validation_acc,\n",
        "                   'training_accuracy', 'validation_accuracy', 'Neural Net on IMDB text classification')\n",
        "\n",
        "with plt.xkcd():\n",
        "    plot_train_val(np.arange(0,epochs), nn_train_loss, nn_validation_loss,\n",
        "                   'training_loss', 'validation_loss', 'Neural Net on IMDB text classification')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "glt7PG16iCKV",
        "cellView": "form"
      },
      "source": [
        "#@markdown Did the neural network perform better or worse than the LSTM from previous week? Why do you think they did so?\n",
        "nn_net_performance = '' #@param {type:\"string\"}"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DRc7VlESthDf"
      },
      "source": [
        "###Exercise 5: LSTM"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Z3ENjdwNthg-"
      },
      "source": [
        "Averaging out the embeddings neglects the sequence of the words in the reviews.\n",
        "For example, consider two sentences:\n",
        "\n",
        "1. our president is a good leader he will not fail \n",
        "2. our president is not a good leader he will fail.\n",
        "\n",
        "Both the sentences will have the same average embeddings. Hence, it is always not a good idea to average the embeddings.\n",
        "\n",
        "Let's implement an LSTM for the same embeddings, similar to what we did last week.\n",
        "\n",
        "We will use similar architecture as the previous week: an LSTM with 2 layers. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KOSeSHptuvuD"
      },
      "source": [
        "class LSTM(nn.Module):\n",
        "  def __init__(self, layers, batch_size, output_size, hidden_size, vocab_size, embedding_length, word_embeddings):\n",
        "    super(LSTM, self).__init__()\n",
        "\n",
        "    self.batch_size = batch_size\n",
        "    self.output_size = output_size\n",
        "    self.hidden_size = hidden_size\n",
        "    self.vocab_size = vocab_size\n",
        "    self.embedding_length = embedding_length\n",
        "    self.n_layers = layers\n",
        "\n",
        "    self.word_embeddings = nn.Embedding(vocab_size, embedding_length)\n",
        "    self.word_embeddings.weight = nn.Parameter(word_embeddings, requires_grad=False)\n",
        "    self.lstm = nn.LSTM(embedding_length, hidden_size, num_layers = self.n_layers)\n",
        "    self.fc1 = nn.Linear(self.n_layers*self.hidden_size, output_size)\n",
        "\n",
        "  def forward(self, input_sentences, batch_size=None):\n",
        "\n",
        "    ####################################################################\n",
        "    # Fill in missing code below (...)\n",
        "    raise NotImplementedError(\"perform the convolution\")\n",
        "\n",
        "    ### Get the word embeddings of the text\n",
        "    ### initialize the hidden layers\n",
        "    ### Pass the embeddinngs through lstm\n",
        "    ### Finally pass the last output through a linear layer\n",
        "    input = ...\n",
        "    hidden = ...\n",
        "\n",
        "    #pass the input through lstm\n",
        "    ...\n",
        "    #get the hidden state of the final sequence\n",
        "    ...\n",
        "    #pass it through the fully connected layers fc1\n",
        "    lstm_output = ...\n",
        "\n",
        "    return lstm_output\n",
        "\n",
        "#to_remove solution\n",
        "  def forward(self, input_sentences, batch_size=None):\n",
        "    input = self.word_embeddings(input_sentences).permute(1, 0, 2)\n",
        "    hidden = (torch.randn(self.n_layers, input.shape[1], self.hidden_size).to(device),\n",
        "            torch.randn(self.n_layers, input.shape[1], self.hidden_size).to(device))\n",
        "    output, hidden = self.lstm(input, hidden)\n",
        "    h_n = hidden[0].permute(1, 0, 2)\n",
        "    h_n = h_n.contiguous().view(h_n.shape[0], -1)\n",
        "    logits = self.fc1(h_n)\n",
        "    return logits"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JoKVKQiUzx62"
      },
      "source": [
        "learning_rate = 0.0003\n",
        "batch_size = 32\n",
        "output_size = 2\n",
        "hidden_size = 32\n",
        "embedding_length = 300\n",
        "epochs = 20\n",
        "word_embeddings = TEXT.vocab.vectors\n",
        "vocab_size = len(TEXT.vocab)\n",
        "layers = 2\n",
        "\n",
        "lstm_model = LSTM(layers, batch_size, output_size, hidden_size, vocab_size, embedding_length, word_embeddings)\n",
        "lstm_model.to(device)\n",
        "lstm_start_time = time.time()\n",
        "lstm_train_loss, lstm_train_acc, lstm_validation_loss, lstm_validation_acc = train(lstm_model, device, train_iter, valid_iter, epochs, learning_rate)\n",
        "print(\"--- Time taken to train = %s seconds ---\" % (time.time() - lstm_start_time))\n",
        "test_accuracy = test(lstm_model, device, test_iter)\n",
        "print('Test Accuracy: ',  test_accuracy, '%')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Wz6IU-49q1sT"
      },
      "source": [
        "with plt.xkcd():\n",
        "    plot_train_val(np.arange(0,epochs), lstm_train_acc, lstm_validation_acc,\n",
        "                   'training_accuracy', 'validation_accuracy', 'LSTM on IMDB text classification')\n",
        "\n",
        "with plt.xkcd():\n",
        "    plot_train_val(np.arange(0,epochs), lstm_train_loss, lstm_validation_loss,\n",
        "                   'training_loss', 'validation_loss', 'LSTM on IMDB text classification')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_LtQKUA0nr54"
      },
      "source": [
        "*Estimated time: 80 minutes since start*"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cjum2sH83ar5"
      },
      "source": [
        "---\n",
        "##Section 4: Attention"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KaGRmX9YRUtt",
        "cellView": "form"
      },
      "source": [
        "#@title Video : Attention\n",
        "from IPython.display import YouTubeVideo\n",
        "video = YouTubeVideo(id=\"VY2Ewm45tO4\", width=854, height=480, fs=1)\n",
        "print(\"Video available at https://youtube.com/watch?v=\" + video.id)\n",
        "\n",
        "import time\n",
        "try: t0;\n",
        "except NameError: t0=time.time()\n",
        "\n",
        "video"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xyEh3inUvdUE"
      },
      "source": [
        "Reference Links: \\\\\n",
        " https://jalammar.github.io/visualizing-neural-machine-translation-mechanics-of-seq2seq-models-with-attention/ \\\\\n",
        "\n",
        " https://d2l.ai/chapter_attention-mechanisms/attention-cues.html\n",
        " "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ceE53uPL9L8v"
      },
      "source": [
        "Last week, we designed an encoder-decoder architecture based on two RNNs for sequence to sequence learning. Specifically, the RNN encoder transforms a variable-length sequence into a fixed-shape context variable, then the RNN decoder generates the output (target) sequence token by token based on the generated tokens and the context variable. However, even though not all the input (source) tokens are useful for decoding a certain token, the same context variable that encodes the entire input sequence is still used at each decoding step. It is challenging for the models to deal with long sentences. \n",
        "\n",
        "In [Bahdanau et al., 2014], the authors proposed a technique called attention. When predicting a token, if not all the input tokens are relevant, the model aligns (or attends) only to parts of the input sequence that are relevant to the current prediction. \n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NjsbJzRSrcGQ"
      },
      "source": [
        "<figure>\n",
        "    <center><img src=https://jalammar.github.io/images/seq2seq_7.mp4>\n",
        "    <figcaption> Sequence to Sequence model with Attention </figcaption>\n",
        "    </center>    \n",
        "</figure>\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FBP08Nwg8DZh",
        "cellView": "form"
      },
      "source": [
        "#@title Run this if above link is broken\n",
        "\n",
        "url = \"https://jalammar.github.io/images/seq2seq_7.mp4\"\n",
        "from IPython.display import HTML\n",
        "HTML(f\"\"\"<video src={url} width=500 controls/>\"\"\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CVSOhlI3AVyN"
      },
      "source": [
        "In contrast to seq2seq model, the encoder passes a lot more data to the decoder. Instead of passing the last hidden state of the encoding stage, the encoder passes all the hidden states to the decoder. \n",
        "\n",
        "In order to focus on the parts of input relevant to the decoder, look at the set of encoder hidden states it received. Each encoder hidden state is at most associated with a certain word in the input sentence. We can assign each hidden state a score and multiply it with the softmaxed score, thus amplifying hidden states with high scores, and drowning out hidden states with low scores."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4ff07bS3tRUB"
      },
      "source": [
        "<figure>\n",
        "    <center><img src=https://jalammar.github.io/images/seq2seq_9.mp4>\n",
        "    <figcaption> Mapping input to output </figcaption>\n",
        "    </center>    \n",
        "</figure>\n",
        "\n",
        "<!-- <video width=\"320\" height=\"240\" controls>\n",
        "      <source src=\"https://jalammar.github.io/images/seq2seq_9.mp4\" type=\"video/mp4\">\n",
        "</video> -->"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OhSS_sGm9ZZx",
        "cellView": "form"
      },
      "source": [
        "#@title Run this if above link is broken\n",
        "\n",
        "url = \"https://jalammar.github.io/images/seq2seq_9.mp4\"\n",
        "from IPython.display import HTML\n",
        "HTML(f\"\"\"<video src={url} width=1000 controls/>\"\"\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Vt3dFpAC95WT"
      },
      "source": [
        "###Queries, Keys, and Values\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8EuBtHTtE-3m"
      },
      "source": [
        "To calculate the attention mechanism we make use of Queries, Keys and Values. But what are Queries, Keys and Values? Query, Value and Key are the transformations of the input vector. \n",
        "\n",
        "In an attention mechanism the context vector is computed as a weighted sum of values, where the weight assigned to each value is computed through an attention score. The score is usually the dot product between the query and key. The scores then go through the softmax function to yield a set of weights whose sum equals 1.\n",
        "\n",
        "The query is from the decoder hidden state whereas the key and value are from the encoder hidden state. \n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7av62SINE_xq"
      },
      "source": [
        "Take a minute and look at this article. It has detailed graphical explanation on how to calculate attention scores. \n",
        "https://towardsdatascience.com/illustrated-self-attention-2d627e33b20a"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8cQMEAv_ps71"
      },
      "source": [
        "### Exercise 6: Attention for Text Classification"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wGkny1-BqmFQ"
      },
      "source": [
        "\n",
        "Until now, we looked at attention aimed at seq2seq networks. Let’s try implementing attention for the above IMDB sentiment analysis dataset. Previously, using the LSTM, the classification completely depended on the last hidden state. In this exercise, we will compute the attention scores between the last hidden state and output of each sequence. The final attention vector will be the weighted average of the outputs at each sequence, with the weights being the attention scores. Lastly, we will concatenate the attention vector and the last hidden state to get the final output.\n",
        "\n",
        "For simplicity's sake, let's implement attention over an LSTM with 1 layer."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gVAm1CWgpwOD"
      },
      "source": [
        "#Code reference: https://github.com/prakashpandey9/Text-Classification-Pytorch/blob/master/main.py\n",
        "\n",
        "class AttentionModel(torch.nn.Module):\n",
        "    def __init__(self, batch_size, output_size, hidden_size, vocab_size, embedding_length, weights):\n",
        "        super(AttentionModel, self).__init__()\n",
        "\n",
        "        \n",
        "        self.batch_size = batch_size\n",
        "        self.output_size = output_size\n",
        "        self.hidden_size = hidden_size\n",
        "        self.vocab_size = vocab_size\n",
        "        self.embedding_length = embedding_length\n",
        "        \n",
        "        self.word_embeddings = nn.Embedding(vocab_size, embedding_length)\n",
        "        self.word_embeddings.weights = nn.Parameter(weights, requires_grad=False)\n",
        "        self.lstm = nn.LSTM(embedding_length, hidden_size)\n",
        "        self.fc1 = nn.Linear(2*hidden_size, output_size)\n",
        "        \n",
        "    def attention_net(self, lstm_output, final_state):\n",
        "\n",
        "        #Implement the attenion net\n",
        "        # Fill in missing code below (...)\n",
        "        raise NotImplementedError(\"perform the convolution\")\n",
        "\n",
        "        \"\"\" \n",
        "        lstm_output : shape: (num_seq, batch_size, hidden_size)\n",
        "        final_state : shape: (1, batch_size, hidden_size)\n",
        "        \"\"\"\n",
        "\n",
        "        # permute the output to get the shape (batch_size, num_seq, hidden_size)\n",
        "        # Get the attention weights \n",
        "        # use torch.bmm to compute the attention weights between each output and hast hidden state\n",
        "        # pay attention to the tensor shapes, you may have to use squeeze and unsqueeze functions\n",
        "        # softmax the attention weights\n",
        "        # Get the new hidden state, use torch.bmm to get the weighted lstm output\n",
        "        # pay attention to the tensor shapes, you may have to use squeeze and unsqueeze functions\n",
        "\n",
        "        lstm_output = ... \n",
        "\n",
        "        attn_weights = ...      #expected shape: (batch_size, num_seq)\n",
        "\n",
        "        soft_attn_weights = ...\n",
        "\n",
        "        new_hidden_state = ...\n",
        "        \n",
        "        return new_hidden_state\n",
        "    \n",
        "    def forward(self, input_sentences):\n",
        "\n",
        "      input = self.word_embeddings(input_sentences)\n",
        "      input = input.permute(1, 0, 2)\n",
        "\n",
        "      h_0 = Variable(torch.zeros(1, input.shape[1], self.hidden_size).cuda())\n",
        "      c_0 = Variable(torch.zeros(1, input.shape[1], self.hidden_size).cuda())\n",
        "\n",
        "      output, (final_hidden_state, final_cell_state) = self.lstm(input, (h_0, c_0))\n",
        "      attn_output = self.attention_net(output, final_hidden_state)\n",
        "      final_output = torch.cat((attn_output, final_hidden_state[0]), 1)\n",
        "      logits = self.fc1(final_output)\n",
        "          \n",
        "      return logits\n",
        "\n",
        "#to_remove solution\n",
        "    def attention_net(self, lstm_output, final_state):\n",
        "\n",
        "        lstm_output = lstm_output.permute(1, 0, 2)\n",
        "        hidden = final_state.squeeze(0)\n",
        "        attn_weights = torch.bmm(lstm_output, hidden.unsqueeze(2)).squeeze(2)\n",
        "        soft_attn_weights = F.softmax(attn_weights, 1)\n",
        "        new_hidden_state = torch.bmm(lstm_output.transpose(1, 2), soft_attn_weights.unsqueeze(2)).squeeze(2)\n",
        "        \n",
        "        return new_hidden_state\n",
        "\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "F5Bnb1qIPP1f"
      },
      "source": [
        "learning_rate = 0.0001\n",
        "batch_size = 32\n",
        "output_size = 2\n",
        "hidden_size = 16\n",
        "embedding_length = 300\n",
        "epochs = 12\n",
        "word_embeddings = TEXT.vocab.vectors\n",
        "vocab_size = len(TEXT.vocab)\n",
        "\n",
        "attention_model = AttentionModel(batch_size, output_size, hidden_size, vocab_size, embedding_length, word_embeddings)\n",
        "attention_model.to(device)\n",
        "attention_start_time = time.time()\n",
        "attention_train_loss, attention_train_acc, attention_validation_loss, attention_validation_acc = train(attention_model, device, train_iter, valid_iter, epochs, learning_rate)\n",
        "print(\"--- Time taken to train = %s seconds ---\" % (time.time() - attention_start_time))\n",
        "test_accuracy = test(attention_model, device, test_iter)\n",
        "print('Test Accuracy: ',  test_accuracy, '%')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PTDsiRS-5KkD"
      },
      "source": [
        "with plt.xkcd():\n",
        "    plot_train_val(np.arange(0,epochs), attention_train_acc, attention_validation_acc,\n",
        "                   'training_accuracy', 'validation_accuracy', 'attention on IMDB text classification')\n",
        "\n",
        "with plt.xkcd():\n",
        "    plot_train_val(np.arange(0,epochs), attention_train_loss, attention_validation_loss,\n",
        "                   'training_loss', 'validation_loss', 'attention on IMDB text classification')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WPilw8NmnymH"
      },
      "source": [
        "*Estimated time: 120 minutes since start*"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gbgTqgB5rH_W"
      },
      "source": [
        "---\n",
        "# Wrap up\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "98YXpSntrY6k",
        "cellView": "form"
      },
      "source": [
        "#@markdown #Run Cell to Show Airtable Form\n",
        "#@markdown ##**Confirm your answers and then click \"Submit\"**\n",
        "\n",
        "import time\n",
        "import numpy as np\n",
        "import urllib.parse\n",
        "from IPython.display import IFrame\n",
        "def prefill_form(src, fields: dict):\n",
        "  '''\n",
        "  src: the original src url to embed the form\n",
        "  fields: a dictionary of field:value pairs,\n",
        "  e.g. {\"pennkey\": my_pennkey, \"location\": my_location}\n",
        "  '''\n",
        "  prefill_fields = {}\n",
        "  for key in fields:\n",
        "      new_key = 'prefill_' + key\n",
        "      prefill_fields[new_key] = fields[key]\n",
        "  prefills = urllib.parse.urlencode(prefill_fields)\n",
        "  src = src + prefills\n",
        "  return src\n",
        "\n",
        "\n",
        "#autofill time if it is not present\n",
        "try: t0;\n",
        "except NameError: t0 = time.time()\n",
        "try: t1;\n",
        "except NameError: t1 = time.time()\n",
        "try: t2;\n",
        "except NameError: t2 = time.time()\n",
        "try: t3;\n",
        "except NameError: t3 = time.time()\n",
        "try: t4;\n",
        "except NameError: t4 = time.time()\n",
        "try: t5;\n",
        "except NameError: t5 = time.time()\n",
        "try: t6;\n",
        "except NameError: t6 = time.time()\n",
        "try: t7;\n",
        "except NameError: t7 = time.time()\n",
        "\n",
        "# autofill fields if they are not present\n",
        "# a missing pennkey and pod will result in an Airtable warning\n",
        "# which is easily fixed user-side.\n",
        "try: my_pennkey;\n",
        "except NameError: my_pennkey = \"\"\n",
        "try: my_pod;\n",
        "except NameError: my_pod = \"Select\"\n",
        "try: last_week_recap;\n",
        "except NameError: last_week_recap = \"\" \n",
        "try: NLP_summary;\n",
        "except NameError: NLP_summary = \"\" \n",
        "try: words_with_opposite_meanings;\n",
        "except NameError: words_with_opposite_meanings = \"\" \n",
        "try: embeddings_improve;\n",
        "except NameError: embeddings_improve = \"\" \n",
        "try: nn_net_performance;\n",
        "except NameError: nn_net_performance = \"\" \n",
        "\n",
        "times = np.array([t1,t2,t3,t4,t5,t6,t7])-t0\n",
        "\n",
        "fields = {\n",
        "    \"my_pennkey\": my_pennkey,\n",
        "    \"my_pod\": my_pod, \n",
        "    \"last_week_recap\": last_week_recap,\n",
        "    \"NLP_summary\": NLP_summary,\n",
        "    \"words_with_opposite_meanings\": words_with_opposite_meanings,\n",
        "    \"embeddings_improve\": embeddings_improve,\n",
        "    \"nn_net_performance\": nn_net_performance,\n",
        "    \"cumulative_times\": times\n",
        "}\n",
        "\n",
        "src = \"https://airtable.com/embed/shrP3tKTTavRNizNY?\"\n",
        "\n",
        "display(IFrame(src = prefill_form(src, fields), width = 800, height = 400))\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oDE0MJbb5dLH"
      },
      "source": [
        "## Feedback\n",
        "How could this session have been better? How happy are you in your group? How do you feel right now?\n",
        "\n",
        "Feel free to use the embeded form below or use this link:\n",
        "<a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://airtable.com/shrNSJ5ECXhNhsYss\">https://airtable.com/shrNSJ5ECXhNhsYss</a>"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IPPjyA-H5kLE"
      },
      "source": [
        "display(IFrame(src=\"https://airtable.com/embed/shrNSJ5ECXhNhsYss?backgroundColor=red\", width = 800, height = 400))"
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}